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• Procedures are critical to 
conduct any complex 
operation 

• Procedures contain knowledge 
about how to operate systems to 
achieve mission goals 


• Procedures are the approved 
means by which a user operates 
a system 

• Users of procedures include 
crew, flight controllers, 
instructors, mission designers, 
payload community, etc. 


Flight Procedure 
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Procedure Requirements 


• Need support for automating procedure execution 

- Commands and telemetry 

- Safety conditions/context 

- Explicit control structures 

• Don’t want to lose human readability 

- Capturing “look-and-feel” of current procedures 

- Presentation of procedure content in a human-friendly way 

• Improve quality of execution 

- Improved ease of use 

- Reduction of human error 

- Improved situational awareness 

• Interleave human actions with automated scripts 

• Use Procedure Representation Language 

- Capture and formalized the above stated requirements 

- Started from NASA ODF standards and construct support automation 
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Procedure Representation 
Language (PRL) file 


Uses of PRL 
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Procedure Viewer & Executor 



TOC A. Buffer Changeout Pro c. 

Siigj 1 - Cover Htmoval 


Siep 3 ■ nutter Removal 
Sie*i 4 - Bulier HejUacemern 


WARNIN'' Remove lid of 
container slowly, as 
contents are under 
pressure and cou!d burst! 


Google Glass - Focus on Mobility & 
mobile interactions 


AR-eProc- Focus on mixed reality interaction 


Capture Rich Procedure Content Once and Use It Everywhere! i 
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Procedure Lifecycle Development 


• Procedure Authoring Tool (PAT) 

- Procedure authors currently use IPV (Licensed 
software & not easy to use) 

- Need an easy-to-use authoring environment 

- Need an easy method to add telemetry & 
commands 

• Procedure verification & validation (PV) 

- Procedure verifiers are human intensive 

- Need for desktop verification tools to catch simple 
mistakes 

• Procedure Library Admin. (PLA) 

- Configuration control works reasonably well 
today 

- Need to be integrated with Procedure Repository 
and Procedure approval system 

• Procedure Viewer/Executor (PVE) 

- Integration with crew time and Caution & 

Warning system 

- Need to view/execute/track anywhere and any 
configuration (stationary, mobile, hand-free. Etc.) 

• Procedure training 

- Integration with Workflow CR and procedure 
verification and validation 

- Measure and track performance 



Measurement 

Performance 
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System Representation 


• Procedure language describes how to operate any system. They do 
not describe the system itself 

• System representation needs to define 

- Telemetry 

- Commands and command parameters 

- System hierarchy and classes 

• e.g., commanding the Orion Display Pages 

• Must be available during procedure editing, validation and 
execution 

• We selected XML Telemetric & Command Exchange (XTCE) — 
an industry and NASA standard 


Johnson Space Center 

Engineering Directorate 

Software, Robotics and Simulation Division 


WebPD 


^ Procedure Log PrcfarencK 
Recommended 


Active 


Ultrasound - Kidnay Seal 



£ fl. I P<1 ) Ifftntwinnd Kidney Scan 

# Start * Stop 

ObfedivB 

To acquire images and measurements of the patient's right and left kidney 
PARTS: 

Ultrasound CX50 
Ultrasound Probe - C5-1 
Probe sc an head cover 

- Probe Connector Cover 
Reference Documents 

- Keyboard and Probe Placement Cue Card 
Ultrasound Echo Get {one bottle]' 

Dry Wipes 


2.01 T61P Device changeouE 

2.02 Hard Drive Not Functioning 

2. i a T61P Device Chang* out Without unstow 
2.1b T61P Device Change out Without Unstow Or St 
■2 s Power On and Login To Laptop 

3.01 Atrium H20 Resupply - Ham 

3.02 Atrium H20 Resupply - Aux 

3.03 Terminate Atrium H20 Resupply 

3.04 Atrium Mam Transfer - Flow High 

3. 05 Atrium Al»: Transfer - Flow High 

3.06 Atrium H20 Resupply With A3 Failed High 

3.07 Atnum Mam Transfer • Flow Low 

3. 05 Atrium Aux Transfer - Row Low 

1.01 Filter Chan geo ut 

1.02 I RED Canister inspection and Cleaning 

5.01 Calf and Bicep Muscle Atrophy and Pam 

3.02 5LM Measurement 

5.03 Ultrasound ■ Configure CX5Q Unit 

5.04 Ultrasound Appendicitis Scan 
j, OS Examination - Abdominal Pam 

5.06 Genitourinary Anatomy 

5.07 Treating Pain 

5. 05 Vital Signs 

5.09 Genitourinary Procedure - unne Retention and 


5.11 ultrasound - Gallbladder Scan 

5.12 Ultrasound ■ Bladder Scan 

5.01 Sample Transfer 

5.02 Plant Soil pH Determination 

2.01 24 VDC Power Supply LED Cheek 

1.02 28 VDC Power Supply Check 

2.03 Wireless Sensor Node Off Line 

2.04 cfltO l 9477 Card Off Line 

2.05 28 V Power supply Failed 

2.06 CREO l Off Line 

2.07 24 V Power Supply Off Line 
3.01 OSH Back Side Camera Survey 

2,0,106 EXAM - PERIODIC HEALTH EVALUATION 


1 CONFIGURE CXfiO Ut TRA SOUND DEVICE 


NOTE 

* The body has two kidneys and they are nol always symmelncal in shape or posmon as a result the operator will have lo 
image and measure both kidneys The right kidney is usually lower In the body by 1-2 Inches and ts easier to find 
i Supine position (lying down) is preferred for I he patient and Ihey can roll to the right and let! lor the right and left kidneys, 
respec lively 


Pi 1 t Configure Itie CX5G Ultrasound Device with Ihe CEM probe using procedure "5 3 Ultrasound - Configure CX5Q 
Device- 

Ill 1 2 For "PreseT on the CX50, select 1 Abd Renar 

PI 1.3 The gel should be applied to applied and resemble whai ts seen tn Figure i 



Figure Ultrasound probe wilh gel applied to Ihe scan head The thumb lying over the indicator notch (red arrow) The gel application m Ihts 
picture (blue arrow) Is correcl for any of the probes that are used tor any ultrasound image application 

| 2 SCANNING PROCEDURE 

H| 2 1 Expose Ihe patient's abdomen 
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WebPD - Automation 


($) 10.02(1) L ight Test - Mozilla Firefo; 

File Edit View History Bookmarks lools H< 
I P 10.02a) Light Test 
localhost 


Procedure Preferences 


Recommended 



Active 


10.02(1) Light Tes 


Completed 


10.02(1) Light Test 


M Start ♦ Stop 

Objective: 

Turn all 8 PDU ports on Bank 1 ON. check that they are turned ON. turn all 8 lights ON, check that they are turned ON. turn all 8 lights 
OFF, check they are all turned OFF, turn all 8 PDU ports on Bank 1 OFF, check that they are turned OFF. 

mm 1 Turn On PDU Ports for Lights Y< 

Q 1.1 cmd [RIU1] POWER_PDU_PORT_ON_CMD 
Q 1.2 cmd [RIU1] POWER_PDU_PORT_ON_CMD 
WM 1.3 cmd [RIU1] POWER_PDU_PORT_ON_CMD 
WM 1.4 cmd [RIU1] POWER_PDU_PORT_ON_CMD 
WM 1.5 cmd [RIU1] POWER_PDU_PORT_ON_CMD 
WM 1.6 cmd [RIU1] POWER_PDU_PORT_ON_CMD 
| D 1.7 cmd [RIU1] POWER_PDU_PORT_ON_CMD 
WM 1.8 cmd [RIU1] POWER_PDU_PORT_ON_CMD 




TOC 



1234 Drive PTZ Camera 
CCPT Control Center Pre-Test Setup 
Fuel LCH4 Fill 

LOW_P Low Press Leak Check 
LOX LOX Fill 
PF Pre-Fill Checklist 
PwrUp Morpheus Power Up 
dsh 

10.0_Normal_Operating 

10.01 Set Controllers to Operating 


10.02 Light Test 


2 Check PDU Ports Are On 

| 2.1 verify [RIU1] POWER_PDU_BANK1_PORT1_ACTUATOR (PORT_ON ^ 
] 2.2 verify [RIU1] POWER_PDU_BANK1_PORT2_ACTUATOR (PORT ON 
] 2.3 verify [RIU1] POWER_PDU_BANK1_PORT3_ACTUATOR ( PORT ON 
] 2.4 verify [RIU1] POWER_PDU_BANK1_PORT4_ACTUATOR ( PORT ON 
] 2.5 verify [RIU1] POWER_PDU_BANK1_PORT5_ACTUATOR ( PORT_ON ^ 
] 2.6 verify [RIU1] POWER_PDU_BANK1_PORT6_ACTUATOR ( PORT ON 
] 2.7 verify [RIU1] POWER_PDU_BANK1_PORT7_ACTUATOR (PORT ON 
1 2.8 verify [RIU1] POWER_PDU_BANK1_PORT8_ACTUATOR (PORT_ON^ 


Enter this step' 


10.02a Light One 
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WebPD - 


Adding Image Notes 
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WebPD - Adding Text Notes 


$ [phe]Periodic Health Examination 


T Fie Settings Views User DR 


Active Procedures 


phe Periodic Health Examina 


4 III 


Completed Procedures 


Table of Contents 


_J 1 Landing Test 
J CCPT Control Center F 
J Fuel LCH4 Fill 
_J LOW_P Low Press Lea 
J LOX LOX Fill 
J 3.PF Pre-Fill Checklist : 

J PTZ_CAM 0 2 1 H_T_ 1 

_J PwrUp Morpheus Pow* * 
J l.PADPF Pad Pre-Fligh 
— 11 umsfiii 

_J 1 SequenceMaster 


[phe] Periodic Health Examination 

• Start 4 Stop 

Fencxnc tteartn examination 

Objective: Perform a general health examination 
1 Collect Patient Vitals 


1 . 1 [CMO] Enter Temperature (F) 


1 .2 [CMO] Enter Blood Pressure 


1.3 [CMO] Enter Pulse 

1 .4 [CMO] Enter Pulse Oximetry 

1.5 [CMO] Enter Respiratory Rate 

1.6 [CMO] Submit Vitals 


2 Skin Examination 


2. 1 [CMO] Launch Webcam Controller. 


‘ Add Note 
Take Picture 


2.2 [CMO] Using webcam to document any rash, injury or other ailments that have developed 




Johnson Space Center 

Engineering Directorate 

Software, Robotics and Simulation Division 


WebPD - IM 
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Medical Exam 
Electronic Procedure 


Use Case 
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EMSD Vision 


The Exploration Medical System Demonstration (EMSD) will 
utilize the ISS as a test bed to demonstrate that several medical 
technologies and medical informatics tools, needed for 
collecting and managing evidence and decision making on an 
exploration mission, can be integrated into a single system and 
used by the on-orbit crew to optimize medical care delivery 
and data management. 
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EMSD 

OPERATIONAL CONCEPT 


Space Center 

ng Directorate 

Robotics and Simulation Division 


EMSD System Overview 
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EMSD Core Components 


Electronic Medical Record 

Medical Data Storage 



* > 

Crewmember 






Assisted Medical Procedures 

User Interface to System 
Procedure Authoring 
Procedure Execution 

Middleware 

Commanding 

Data Transfer Brokering 

Scalability 

Clinical Decision Support 

Future Development 


Peripheral Devices 



Vehicle Computer 
(SSC) 


r 

Assisted Medical 

_J 

Clinical Decision 

Procedures 


Support 

i / 


HJ 


Middleware 


Electronic 
Medical Record 
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EMSD Peripheral Devices 



Ultrasound (U/S) 

Data File Import 
Streaming Video 

Inventory Tracker 

Data File Import 

Webcam + Dental Camera 

Streaming Video 
Image/Video File 

Biosensors (ECG) 

Commanding 
Data File Import 

Crewmember 

Data Entry 
Navigation 


Peripheral Devices 



d 


E 


1 


© 


\ 


u/s 


RFID Webcam Dental Biosensors 
Scanner Camera 


Vehicle Computer 
(SSC) 


Companion 

Software 


EMSD Core Components 


rO 


j 
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EMSD 

SYSTEM DESIGN 
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Assisted Medical 
Procedures 

Electronic Medical Record 

Middleware 


Image/Video Capture 


Component Selection 


Procedure Authoring Tool 
Web Procedure Display (WebPD) 

OpenEMR/CouchDB 

OpenDDS 


HTML5/VSee 
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Methods for Data Integration 
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EMSD Architecture Overview 


Mtdlca 
^Eccrr& DE 


■DDS Top-b" 


File access 



Fie 

T ra nefflp 

1 k_l 


file 

^^1 1 lilllaln 

V Software j 


file 


Electro me 
Medical Records 


Web 

Services 


-Query ■' Update- 


JL 

IP 


Files 


Notifies tic 


Notifications 


Files 



DOS 


Notifications Fi es 


DDS 

Adapter 

J 

} 

1 

r 

f EC 
I Softi 

;g \ 

#a r e J 


Login 


DDS 

Adapter 


file 


/" Video 
Conferencing 
'v Software 


HDMI 
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physical Implementation - Single Host 
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Software 
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Physical Implementation - Multiple Hosts 


file 
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File access 
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What's next? 
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Evolution of Procedures 





T ■ !■ li ■ !■ li i 



Early ISS— PDF 


Current ISS— IPV/XML 


Deep Space Exploration- AR 
• PRL Extension 


Orion; Enhanced XML (PRL) 

• Computer Oversight 

• Automation 


XT A . • Machine Vision and Marker-less Registration 

No Automation or Computer ° 

Oversight 
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Future Direction: Autonomous Ultrasound Ops. 




1. Image detection software depiction of 2. Robonaut 2 being remotely guided through carotid 

anatomical landmarks which define an artery ultrasound imaging technique 

adequate carotid image superimposed over 
an actual carotid artery ultrasound image 
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Backup 
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SEATEST II: T ust-in-time training & procedure execution 

with Glass 


Miniature Exercise Device (MED): 

a. Equipment Assembly Task 

b. Equipment Dis- Assembly Task 



Just-in-time (JIT) training of a Sani-tank 
purge 

After the task was completed using the 
Google Glass - the same JITT material was 
viewed on an iPad 


r 


Return 
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Augmented Reality (AR-eProc ARED) 












